Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.32%. Comparing base (d475be0) to head (84cf94e).
⚠️ Report is 2913 commits behind head on main.

@@             Coverage Diff              @@
##               main    #5842      +/-   ##
============================================
+ Coverage     89.28%   92.32%   +3.03%     
+ Complexity     6585     6555      -30     
============================================
  Files           693      699       +6     
  Lines         19937    19774     -163     
============================================
+ Hits          17801    18256     +455     
+ Misses         2136     1518     -618     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@KentarouTakeda could you please double check your reported issue still exist?
I have a feeling this was fixed in the meantime.

if it still does not produce the expected outcome, please provide describe in more detail why/what does not work.

thank you

@staabm
The issue still persists. I tested with the two latest actively supported versions, 11.5.33 and 12.3.5, and confirmed the same behavior as in the initial report.

Let me explain the details. Please take a look at the test code I first presented. test1 and test3 contain exactly the same code. Nevertheless, test1 passes while test3 fails. In this case, the success of test1 is the correct behavior. The unintended failure of test3 is caused by the failure of test2 with expectOutputString(). This pull request aims to fix these behaviors.

The minimal reproduction code above demonstrates the issue, but this behavior also affects feature tests in many frameworks, including Laravel and Symfony, producing unintended test result reports for users.

For example, in Laravel, running the following test code:

<?php

namespace Tests\Feature;

use PHPUnit\Framework\Attributes\DataProvider;
use PHPUnit\Framework\Attributes\Test;
use Tests\TestCase;

// Important note: Here, `TestCase` extends the Laravel Framework's `TestCase`.
class ExampleTest extends TestCase
{
    #[Test]
    public function intentionalFailure()
    {
        $this->expectOutputString('This test will be failed'); // Failure (intentional)
    }

    #[Test]
    #[DataProvider('repeatProvider')]
    public function tenSuccessfulTests()
    {
        $this->assertTrue(true);
    }

    public static function repeatProvider()
    {
        return array_map(fn() => [], range(1, 10));
    }
}

produces the following result:

FRRRRRRRRRR                                                       11 / 11 (100%)

While the first test fails as expected, it is not expected that all subsequent tests are marked as Risky.

The detailed output includes messages like:

There was 1 risky test:

1) Tests\Feature\ExampleTest::tenSuccessfulTests
* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

As shown above, in frameworks that manipulate error or exception handlers (including Laravel and Symfony), a failure in tests involving input/output can lead to user confusion due to misleading test reports.

This pull request fixes the problem where the failure of one test causes handler state inconsistencies that propagate to subsequent tests, resulting in them being incorrectly marked as Risky.